In order to provide better access to large document collections, it is desirable to develop automatic summarizers that will produce summaries of each document or a summary of a cluster of documents. One obstacle to developing such summarizers is that it is difficult to evaluate the quality of the summaries produced by the automatic summarizers, and therefore it is difficult to train the summarizers.
One ad-hoc technique for evaluating automatic summaries involves determining how many words found in a manually-created summary are also found in the automatic summary. The number of words found in the automatic summary is divided by the total number of words in the manual summary to provide a score for the automatic summary. This ad-hoc measure is less than ideal because there is no theoretical justification for believing that it would provide scores that correlate to the quality of a summary. In fact, it has been observed that summaries that receive a poor score using this ad-hoc measure are often judged to be good summaries when evaluated by a person.
The discussion above is merely provided for general background information and is not intended to be used as an aid in determining the scope of the claimed subject matter.